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- The MAILING DATE of this communication appears on the cover sheet with the correspondence address - 
Period for Reply 

A SHORTENED STATUTORY PERIOD FOR REPLY IS SET TO EXPIRE 3 MONTH(S) FROM 
THE MAILING DATE OF THIS COMMUNICATION. 

- Extensions of time may be available under the provisions of 37 CFR 1 .136(a). In no event, however, may a reply be timely filed 
after SIX (6) MONTHS from the mailing date of this communication. 

• If the period for reply specified above is less than thirty (30) days, a reply within the statutory minimum of thirty (30) days will be considered timely. 

- If NO period for reply is specified above, the maximum statutory period will apply and will expire SIX (6) MONTHS from the mailing date of this communication. 

- Failure to reply within the set or extended period for reply will, by statute, cause the application to become ABANDONED (35 U.S.C. § 133). 

- Any reply received by the Office later than three months after the mailing date of this communication, even if timefy filed, may reduce any 
earned patent term adjustment. See 37 CFR 1.704(b). 

Status 

1)^ Responsive to communication(s) filed on 29 May 2003 . 
2a)Kl This action is FINAL. 2b)D This action is non-final. 

3) D Since this application is in condition for allowance except for formal matters, prosecution as to the merits is 

closed in accordance with the practice under Ex parte Quayle, 1935 CD. 11, 453 O.G. 213. 
Disposition of Claims 

4) ^ Claim(s) 7-30 is/are pending in the application. 

4a) Of the above claim(s) is/are withdrawn from consideration. 

5) D Claim(s) is/are allowed. 

6) ^ Claim(s) 1-30 is/are rejected. 

7) D Claim(s) is/are objected to. 

8) D Claim(s) are subject to restriction and/or election requirement. 

Application Papers 

9) ^ The specification is objected to by the Examiner. 

10) D The drawing(s) filed on is/are: a)Q accepted or b)D objected to by the Examiner. 

Applicant may not request that any objection to the drawing(s) be held in abeyance. See 37 CFR 1.85(a). 

11) ^ The proposed drawing correction filed on 28 May 2003 is: a)^ approved b)D disapproved by the Examiner 

If approved, corrected drawings are required in reply to this Office action. 

12) D The oath or declaration is objected to by the Examiner. 
Priority under 35 U.S.C. §§119 and 120 

13) D Acknowledgment is made of a claim for foreign priority under 35 U.S.C. § 119(a)-(d) or (f). 

a)DAII b)D Some*c)D None of: 

1 .□ Certified copies of the priority documents have been received. 

2. D Certified copies of the priority documents have been received in Application No. . 

3. D Copies of the certified copies of the priority documents have been received in this National Stage 

application from the International Bureau (PCT Rule 17.2(a)). 
* See the attached detailed Office action for a list of the certified copies not received. 

14) D Acknowledgment is made of a claim for domestic priority under 35 U.S.C. § 119(e) (to a provisional application). 

a) □ The translation of the foreign language provisional application has been received. 

1 5) D Acknowledgment is made of a claim for domestic priority under 35 U.S.C. §§ 1 20 and/or 1 21 . 

Attachment(s) 

1 ) (3 Notice of References Cited (PTO-892) 4) □ Interview Summary (PTO-41 3) Paper No(s). . 

2) CJ Notice of Draftsperson's Patent Drawing Review (PTO-948) 5) Q Notice of Informal Patent Application (PTO-152) 

3) □ Information Disclosure Statement(s) (PTO-1449) Paper No(s) . 6) □ Other: 
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DETAILED ACTION 

This action is in response to communication filed 4/29/03. 

Drawings 

1 . The proposed drawing correction and/or the proposed substitute sheets of drawings, filed 
on 4/29/03 have been approved. A proper drawing correction or corrected drawings are required 
in reply to the Office action to avoid abandonment of the application. The correction to the 
drawings will not be held in abeyance. 

Specification 

2. The objection to the specification for lacking a "Summary of the Invention" is 
maintained. Applicant is directed to 37 CFR 1.73, which states, "A brief summary of the 
invention indicating its nature and substance, which may include a statement of the object of the 
invention, should precede the detailed description. . ." It is noted that the brief summary should 
be inserted prior to the detailed description. Applicant is directed to MPEP § 608.01(d). 

Claim Rejections - 35 USC § 112 

3. The following is a quotation of the second paragraph of 35 U.S.C. 1 12: 

The specification shall conclude with one or more claims particularly pointing out and distinctly claiming the 
subject matter which the applicant regards as his invention. 

4. Claims 1-13 and 29-30 are rejected under 35 U.S.C. 112, second paragraph, as being 
indefinite for failing to particularly point out and distinctly claim the subject matter which 
applicant regards as the invention. 
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5. Claim 1 recites, "a specified timing within a gesture will occur", which is unclear. It is 
believed that —which— should be inserted between "within" and "a". 

6. Claim 2 recites "determining a probability that each of a plurality of predefined gestures 
which are performed in the video clip contains the predefined gesture", which essentially states 
determining a probability that a plurality of predefined gestures contains the predefined gesture, 
which does not make sense and renders the claim indefinite. It is unclear as to whether the 
probability of the video clip containing a predefined gesture is being determined or if each of the 
gestures performed in the video clip are predefined and the probability that a specific predefined 
gesture is performed among the plurality of predefined gestures is being determined. 

7. Claim 9 recites the limitation "each video frame" in line 2. There is insufficient 
antecedent basis for this limitation in the claim. 

8. Claim 1 1 recites the limitation "the target gesture" in lines 2-3. There is insufficient 
antecedent basis for this limitation in the claim. 

9. Claim 13 recites the limitation "the predefined gesture" in line 2. There is insufficient 
antecedent basis for this limitation in the claim. 

10. Claims 29-30 recite the limitation "the video processing system" in line 1 . There is 
insufficient antecedent basis for this limitation in the claim. 

Claim Rejections - 35 USC §102 

11. The following is a quotation of the appropriate paragraphs of 35 U.S.C. 102 that form the 
basis for the rejections under this section made in this Office action: 

A person shall be entitled to a patent unless - 
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(e) the invention was described in (1) an application for patent, published under section 122(b), by another filed 
in the United States before the invention by the applicant for patent or (2) a patent granted on an application for 
patent by another filed in the United States before the invention by the applicant for patent, except that an 
international application filed under the treaty defined in section 35 1(a) shall have the effects for purposes of this 
subsection of an application filed in the United States only if the international application designated the United 
States and was published under Article 2 1 (2) of such treaty in the English language. 

12. Claims 1-3, 8-10, 13-16, and 26 are rejected under 35 U.S.C. 102(e) as being clearly 
anticipated by U.S. Patent No. 6,256,033 to Nguyen (hereinafter Nguyen). 

Referring to claims 1 and 26, Nguyen discloses a method and a computer program 
product for recognizing gestures contained in video data that segments video data to create a 
video clip based on timing data that indicates a specified timing within which a gesture will 
occur. See col. 2, lines 23-25, which states, "The frame captures the person in the action of 
performing the gesture at one moment in time" and col. 2, lines 28-34, which states, "These 
sequence of frame data sets taken over a period of time. . The method determines information 
related to a gesture occurring in the video clip only at the specified timing. See col. 3, lines 1 1- 
14, which describes "determining particular coordinates of the subject at a particular time ". 
Also, see col 5, lines 13-30, which describes how the Hidden Markov Model is used to 
determine the most probable state (most likely gesture). 

Referring to claim 14, Nguyen discloses a temporal segmentor connected to receive video 
data and to create a video clip from the video data based on timing data that indicates a specified 
timing within which a gesture will occur (col. 2, lines 23-34 and col. 3, lines 1 1-14). 

Nguyen also discloses a recognition engine, in communication with the temporal 
segmentor, to determine if the video clip contains a predefined gesture, only at the specified 
timing. See col. 1, lines 17-23 and col. 3, lines 11-14. 
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Referring to claims 2-3, col. 5, lines 13-45 of Nguyen describes how Hidden Markov 
Models are used to determine a probability that each of a plurality of predefined gestures are 
performed in the video clip. 

Referring to claims 8-10, each video clip of Nguyen contains video frames (col. 7, line 
22), and in each frame, the moving regions are identified (col. 8, lines 58-61). Feature vectors 
(array of key points) are generated for each video frame of the video clip. See col. 8, line 49 - 
col. 9, line 46, which describes how significant positional coordinates are extracted from each 
frame to make a comparison to the known gesture coordinates. 

Referring to claims 13 and 22, Nguyen discloses that the recognition engine is configured 
to recognize predefined gestures and that determining if the video clip contains a predefined 
gesture includes generating a gesture probability vector (array) having a plurality of elements, 
each element being associated with one of a predefined gestures and representing a probability 
that the video clip contains each of the associated predefined gestures. See col. 1 1, lines 30-49 
and col. 10, lines 18-37. 

Referring to claim 15, the recognition engine of Nguyen includes a plurality of Hidden 
Markov Models. See col 5, lines 13-18. 

Referring to claim 16, Nguyen teaches a timing data source, in communication with the 
temporal segmentor, to provide the timing data. See col. 2, lines 24-3 1, which implies a timing 
data source, because the frames are in data sets for a given period of time. Nguyen shows a 
video source (camera; Fig. 2, 200), in communication with the temporal segmentor, to provide 
the video data to the temporal segmentor. 
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Claim Rejections - 35 USC § 103 



13. The following is a quotation of 35 U.S.C. 103(a) which forms the basis for all 
obviousness rejections set forth in this Office action: 

(a) A patent may not be obtained though the invention is not identically disclosed or described as set forth in 
section 102 of this title, if the differences between the subject matter sought to be patented and the prior art are 
such that the subject matter as a whole would have been obvious at the time the invention was made to a person 
having ordinary skill in the art to which said subject matter pertains. Patentability shall not be negatived by the 
manner in which the invention was made. 

14. Claims 4-7, 1 1-12, 17-21, 23-25, and 27-30 are rejected under 35 U.S.C. 103(a) as being 
unpatentable over Nguyen and U.S. Patent No. 6,227,968 to Suzuki et al. (hereinafter Suzuki). 

Referring to claims 4-6, 23-25, and 27, Nguyen teaches recognizing gestures within a 
specific timing, supra, but Nguyen fails to teach that the timing data includes beat data 
corresponding to a beat of audio data. However, Suzuki teaches a dance game, which is intended 
to make the player use his entire body to create rhythm sensations (col. 1, lines 47-50). Suzuki 
teaches a step-on base (col. 1, lines 54-58) that is used to capture the player's steps, thus 
recognizing the player's gestures (dance moves). Suzuki teaches combining gesture recognition 
with audio beat data. The dance game of Suzuki teaches that audio data is received and the beat 
data is extracted to create timing data (col. 7, lines 24-34). Suzuki teaches a predefined time 
window surrounding the occurrence of at least one beat. See col. 7, lines 35-50. Also, see col. 1, 
lines 9-13, which describes how the player is to perform the action in time with the rhythm. It 
would have been obvious to one of ordinary skill in the art at the time of the invention to modify 
the gesture recognition method of Nguyen to be used in the dance game of Suzuki, such that the 
timing data includes beat data extracted from audio data and where the video clip covers a 
predefined time window surrounding at least one beat as supported by Suzuki. One would have 
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been so motivated in order to compare all movements of the player (i.e. arms) to appropriate 
dance moves because the step-on base of Suzuki only captures the movements of the player's 
feet, and does not recognize the movements of the rest of the body as intended in Suzuki. 

Referring to claims 7, 17-18, and 21, although Nguyen describes predefined gestures, 
Nguyen does not explicitly teach a move subsystem, in communication with the timing data 
source, to display a target gesture to be performed by the subject. However, the dance game of 
Suzuki teaches displaying a target gesture to be performed by the subject. The target gesture is a 
dance move. See col. 12, lines 57-61. The target gesture is displayed on the display subsystem 
(monitor; col. 12, line 59). It would have been obvious to one of ordinary skill in the art at the 
time of the invention to modify the gesture recognition method of Nguyen to display a target 
gesture in order to guide the subject though a dance as supported by Suzuki. 

Referring to claims 11-12 and 19-20, the method of Nguyen and Suzuki, supra, generates 
and displays a score based on whether a target movement (gesture) was performed. See col. 1 1, 
lines 37-41. The object of the Suzuki dance game is to achieve the highest score by performing 
the appropriate gestures. 

Referring to claim 28, Nguyen teaches an A/V processing system that includes a video 
source (camera; Fig. 2, 200). Nguyen discloses a computer program product for recognizing 
gestures contained in video data, comprising instruction operable to cause a programmable 
processor, in communication with the video source to segment the video clip and determine if the 
video clip contains a predefined gesture. See col. 1, lines 17-23. 

Nguyen does not explicitly teach an audio source from which beat data is extracted, such 
that the timing of the segments are based on the beat data. However, Suzuki teaches a dance 
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game, which is intended to make the player use his entire body to create rhythm sensations (col. 
1, lines 47-50). The dance game of Suzuki teaches that audio data is received and the beat data 
is extracted to create timing data (col. 7, lines 24-34). Suzuki determines if a predefined gesture 
(dance move) is performed by detecting dance steps on a step-on base only within a specified 
timing related to the beat data. For example, see col. 14, lines 6-18. It would have been obvious 
to one of ordinary skill in the art at the time of the invention to modify the gesture recognition 
method of Nguyen to be used in the dance game of Suzuki, such that the timing data includes 
beat data extracted from audio data as supported by Suzuki. One would have been so motivated 
in order to compare all movements of a player (i.e. arms) to an appropriate dance move because 
the step-on base of Suzuki only captures the movements of the player's feet, and does not 
recognize the movements of the rest of the body as intended in Suzuki. 

Referring to claim 29, Nguyen teaches the computer program product includes 
instruction operable to cause the programmable processor to perform a Hidden Markov Model 
process to determine if the video clip contains the predefined gesture. See col. 5, lines 13-18. 

Referring to claim 30, Nguyen discloses a display (monitor; Fig. 2, 208) to display 
information based on whether the video clip contains the predefined gesture. See col. 6, lines 
36-44, which describes how figures on the display can be augmented if the gesture is recognized 
as a predefined gesture. 

Response to Arguments 
15. Applicant's arguments filed 4/29/03 have been fully considered but they are not 
persuasive. 
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Applicant argues that since Nguyen teaches a real-time gesture recognition system that 
the gestures are not recognized only at a specified timing. However, Nguyen describes that it is 
the comparisons between the data derived from the input frame and data previously derived that 
is done in real-time. See col. 2, lines 12-16. Nguyen does teach determining information 
relating to a gesture only at a specified timing. See col. 2, lines 23-25, which states, "The frame 
captures the person in the action of performing the gesture at one moment in time" and col. 2, 
lines 28-34, which states, "These sequence of frame data sets taken over a period of time. . ." 
Also, see col. 3, lines 11-14, which describes "determining particular coordinates of the subject 
at a particular time ". Applicant further argues that adding timing information into the gesture 
recognition provides a special advantage that enables the system to be used advantageously, but 
does not argue what the advantage is. It would be impractical for Nguyen not to use timing data 
to limit the amount of frames to be analyzed for determining the gesture, and as stated above, 
Nguyen teaches segmenting the video data according to time periods, and thus teaches each 
limitation in claims 1-3, 8-10, 13-16, and 26. 

It should be noted that Applicant admits in the background of the specification, filed 
9/15/00, that a general approach to gesture recognition is to capture "video data of a user 
performing various actions, temporally segmenting the video into video clips containing discrete 
gestures, and then determining if each video clip contains a predefined gesture from a gesture 
vocabulary." See page 2, lines 14-18. 

16. In response to applicant's argument that the examiner's conclusion of obviousness is 
based upon improper hindsight reasoning, it must be recognized that any judgment on 
obviousness is in a sense necessarily a reconstruction based upon hindsight reasoning. But so 
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long as it takes into account only knowledge which was within the level of ordinary skill at the 
time the claimed invention was made, and does not include knowledge gleaned only from the 
applicant's disclosure, such a reconstruction is proper. See In re McLaughlin, 443 F.2d 1392, 
170 USPQ 209 (CCPA 1971). 

Applicant argues that Suzuki teaches nothing about determining gestures within specified 
beats of the audio. However, the step-on base of Suzuki detects if a player performs the right 
step, which is a type of gesture. See the Abstract, col 1, lines 44 - 58, and col. 7, lines 24-34. 
The purpose of the game in Suzuki is to perform the appropriate dance step (gesture) within each 
beat of audio, thus it was known at the time of the invention to determine gestures within 
specified beats of audio. 

Suzuki teaches a more sophisticated type of gesture recognition capable of being applied 
to the dance game of Suzuki, which would have been known to one of ordinary skill in the art at 
the time of the invention, especially because Suzuki suggests the player should use his entire 
body to create rhythm sensations (col. 1, lines 47-50). 

It should be noted that in the background of the specification, filed 9/15/00, Applicant 
describes that a pressure sensitive pad in a dance game is a recognized specialized input device 
that recognizes input actions driven by a particular human gesture. See page 1, line 14 - page 2, 
line 6. 

Conclusion 

17. THIS ACTION IS MADE FINAL. Applicant is reminded of the extension of time 
policy as set forth in 37 CFR 1 .136(a). 
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A shortened statutory period for reply to this final action is set to expire THREE 
MONTHS from the mailing date of this action. In the event a first reply is filed within TWO 
MONTHS of the mailing date of this final action and the advisory action is not mailed until after 
the end of the THREE-MONTH shortened statutory period, then the shortened statutory period 
will expire on the date the advisory action is mailed, and any extension fee pursuant to 37 
CFR 1 .136(a) will be calculated from the mailing date of the advisory action. In no event, 
however, will the statutory period for reply expire later than SIX MONTHS from the mailing 
date of this final action. 

1 8. The prior art made of record on form PTO-892 and not relied upon is considered 
pertinent to applicant's disclosure. Applicant is required under 37 C.F.R. § 1.1 1 1(c) to consider 
these references fully when responding to this action. The documents cited therein teach a dance 
game system for recognizing gestures within a timing related to beat data. 

Any inquiry concerning this communication or earlier communications from the 
examiner should be directed to Shawn M. Becker whose telephone number is 703-305-7756. 
The examiner can normally be reached on M-Th 8:00 - 5:30 and alternating Fridays. 

If attempts to reach the examiner by telephone are unsuccessful, the examiner's 
supervisor, John W. Cabeca can be reached on 703-305-3 116. The fax phone numbers for the 
organization where this application or proceeding is assigned are 703-746-7239 for regular 
communications and 703-746-7238 for After Final communications. 

Any inquiry of a general nature or relating to the status of this application or proceeding 
should be directed to the receptionist whose telephone number is 703-305-3900. 




JOHN CABECA 
SUPERVISORY PATENT EXAMINER 
TECHNOLOGY CENTER 2100 



Application/Control Number: 09/662,679 
Art Unit: 2173 



smb 

July 21, 2003 



